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[57] ABSTRACT 

A telerobot system including an operator interface for 
developing and controlling complex tasks for execution 
on a remote robot manipulator by developing sequences 
of parameterized tasks primitives for execution and 
testing on a local simulator. The task sequences, and 
their parameterization, may be interactively modified 
by teleoperation and/or editing during sequence devel- 
opment and task execution. Adjustable time delays are 
provided for execution simulation including delays ex- 
pected to occur between task command and execution 
at the remote robot manipulator. 

7 Claims, 1 Drawing Sheet 
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TELEROBOT CONTROL SYSTEM 

ORIGIN OF THE INVENTION 

The invention described herein was made in the per- 5 
formance of work under a NASA contract, and is sub- 
ject to the provisions of Public Law 96-517 (35 USC 
202) in which the Contractor has elected not to retain 
title. 

1. Technical Field 10 

The present invention relates to robotic control sys- 
tems and, in particular, to telerobot systems for use in 
poorly modeled environments, such as those encoun- 
tered in space and undersea projects. 

2. Background of the Invention 15 

Techniques for the operation and control of robotic 

manipulators in well modeled environments, such as 
manufacturing facilities, have been developed and used 
very successfully for many years. Such environments 
may be well modeled because the environment of the 20 
robotic work area can be well known or designed to 
minimize differences between the actual work area and 
the model used by the operation and control techniques. 

These techniques must be substantially modified for 
use in poorly modeled environments, or to perform 25 
tasks in response to unplanned scenarios and when there 
is a potentially substantial time delay between teleoper- 
ation inputs and task execution as occurs for example in 
surface controlled undersea and ground controlled 
space robotic operations. 30 

Remote control of robotic operation may be accom- 
plished by teleoperation, autonomous or supervisory 
control as well as a combination of these approaches 
which is known as shared control. 

Interactive robotic task planning, execution and mon- 35 
itoring can be accomplished with pure teleoperation. In 
this approach, planning resides within the operator’s 
mind, execution is issued by the operator via hand con- 
trollers and monitoring is provided by sensory feedback 
to the operator. Autonomous task planning, execution, 40 
and monitoring is the other extreme to teleoperation. 
Here, the operator initiates only very high level com- 
mands such as “replace the electronics module” and 
planning, execution, and monitoring is then done auton- 
omously without further operator input. 45 

Teleoperation has proven to be a valuable tool for 
many tasks especially in unmodeled or poorly modeled 
environments and for unplanned scenarios. The increas- 
ing complexity of the tasks to be performed places an 
ever increasing burden on the operator. Autonomous 50 
control is becoming increasingly valuable as a tool to 
relieve the operator of many task planning, execution, 
and monitoring responsibilities in order to allow the 
operator to concentrate on the more crucial elements of 
a task. 55 

Supervisory and shared control are recent improve- 
ments in telerobot task execution for unplanned scenar- 
ios, or for poorly modeled environments. Supervisory 
control is where the operator selects execution parame- 
ters for autonomous execution of a task and can stop 60 
execution at any time. Shared control is the mixing of 
inputs from an operator and an autonomous control 
system during task execution. 

A key element needed for planning, execution, and 
monitoring in a supervisory or shared control system is 65 
an operator interface which supports shared and super- 
visory control features. Supervisory features are re- 
quired to permit the operator to set up teleoperation, 
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autonomous, and shared control task environment pa- 
rameters and to provide specific input parameters for 
autonomous task primitives and teleoperation control. 
A task primitive is a reusable, predetermined, self con- 
tained, preprogrammed program for controlling a robot 
to accomplish a task, such control being dependent 
upon a set of input parameters which may be designated 
before or at the time of task execution. Shared features 
of an operator interface are required in order to provide 
autonomous setting of some environment and control 
parameters depending upon task context. 

What is needed is an operator interface for planning, 
execution and monitoring of telerobot tasks in poorly 
modeled environments and for unplanned scenarios 
which efficiently and conveniently combines teleopera- 
tion, autonomous, supervisory, and shared control tech- 
niques. 

BRIEF STATEMENT OF THE INVENTION 

The preceding and other shortcomings of the prior 
art are addressed and overcome by the present inven- 
tion that provides, in a first aspect, a method for con- 
trolling a telerobot by providing a set of task primitives 
to control a local telerobot simulator and a remote 
telerobot manipulator, interactively developing and 
modifying the sequencing and parameterization of a 
sequence of said task primitives executed on the local 
telerobot simulator, and then executing the sequence to 
control the remote telerobot manipulator. 

These and other features and advantages of this in- 
vention will become further apparent from the detailed 
description that follows which is accompanied by a set 
of drawing figure(s). In the figures and description, 
numerals indicate the various features of the invention, 
like numerals referring to like features throughout both 
the drawings and the description. 

BRIEF DESCRIPTION OF THE DRAWING(S) 

FIG. 1 is a block diagram schematic of a telerobot 
control system including an operator interface accord- 
ing to the present invention. 

DETAILED DESCRIPTION OF THE 
INVENTION 

Referring now to FIG. 1, telerobot control system 10 
includes operator interface 12 and remote task execu- 
tion system 14. 

Remote task execution system 14 includes robot ma- 
nipulator 16, suitable for the tasks to be performed, and 
communication link 18 for providing bidirectional com- 
munication with operator interface 12. Robot manipula- 
tor 16 includes one or more remotely controllable de- 
vices such as robot arm 20, end effector 21 and remote 
sensors, such as force-torque sensor 19 and video cam- 
era 22. Communication link 18 includes communication 
and control system processor 24 as well as a suitable 
linkage device, such as antenna 26. 

Operator interface 12 includes one or more operator 
terminals, such as setup, simulation and execution termi- 
nals 28, 30, and 32. It may be convenient to provide 
separate terminals for these tasks as shown, or combine 
the terminals into a single type of multipurpose terminal 
for each operator, capable of performing all required 
functions. 

Operator interface 12 also includes operator interface 
communication processor 34, including a suitable link- 
age device such as antenna 36, as well as graphics simu- 
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lator 38 and local robot manipulator 40 which will be 
described below in greater detail. 

Each operator terminal, such as execution terminal 
32, includes monitor 42, central processor 44, keyboard 
entry device 46, and a hand control and feedback de- 5 
vice, such as hand controller 42. During the various 
operations performed at the operator terminals, the 
operator enters information into the appropriate termi- 
nal via keyboard entry device 46 and/or hand control- 
ler 42 and receives information from remote task execu- 1° 
tion system 14, graphics simulator 38, and/or local 
robot manipulator 40 via monitor 42 and/or hand con- 
troller 48, as appropriate to the task. 

All the devices in operator interface 12 are intercon- 
nected by a conventional interconnection system, desig- 15 
nated generally as interconnection system 50. 

Telerobot control system 10 is operable in a conven- 
tional telerobot control mode in which pure teleopera- 
tion control is employed. During such operation, an 
operator working at execution terminal 32 is directly 2 
connected in real time, or near real time, to remote task 
execution system 14 via operator interface communica- 
tion processor 34. Commands entered via keyboard 
entry device 46 and/or hand controller 48 are imple- ^ 
mented by robot manipulator 16 while information is 
made available to the operator from remote sensors, 
such as force-torque sensor 19 and/or video camera 22, 
via monitor 42 and/or hand controller 48. 

Telerobot control system 10 is operable in a conven- 3Q 
tional autonomous mode in which autonomous control 
is employed. During such control, the operator applies 
an input to execution terminal 32 to cause the execution 
of a preprogrammed task by robot manipulator 16. The 
input is applied by interconnection system 50 and opera- 35 
tor interface communication processor 34 to communi- 
cation link 18 which initiates the execution of the pre- 
programmed task within communication and control 
system processor 24 by robot manipulator 16. Some 
information relating to the status of the task and robot ^ 
manipulator 16 may be collected by sensors, such as 
video camera 22 , and communication and control sys- 
tem processor 24 and be provided to the operator by 
monitor 42. 

Telerobot control system 10 is operable in a conven- 45 
tional supervisory mode in which autonomous task 
execution is initiated by the operator by the provision of 
the appropriate parameters to communication and con- 
trol system processor 24 via keyboard entry device 46 
and/or hand controller 48. The preprogrammed task 50 
within communication and control system processor 24 
initiated in this mode may be in the form of a task primi- 
tive, as described above, which requires the provision of 
a set of input parameters. These parameters may be 
modified during execution. 55 

Remote task execution system 14, in addition to a set 
of task primitives, includes an executive program simi- 
lar to executives within graphics simulator 38 and local 
robot manipulator 40. These executives control one or 
more central processing units, or CPU’s, within remote 60 
task execution system 14 and/or operator interface 12, 
which sequentially call, operate and execute task primi- 
tives, provide status and sensor feedback to operator 
interface 12 and run other programs, in sequence or in 
parallel with the task primitives, such as embedded 65 
safety and monitoring programs. The executive also 
maintains and uses a database of global parameters 
which are normally changed less frequently than the 
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parameters provided to a task primitive would be 
changed. 

The use of a set of task primitives common to opera- 
tor interface 12 and remote task execution system 14 
permit the interactive development and control of the 
operation of tasks by robot manipulator 16 by passing 
only parameters and teleoperation inputs between oper- 
ator interface 12 and remote task execution system 14. 
Control of remote task execution system 14 by parame- 
trization which may be developed and tested at a local 
site before being applied to remote task execution sys- 
tem 14 permits a relative wide range of operation of 
robot manipulator 16 without the transmission of new 
programming from the local to the remote site. The 
sequence of parameterized task primitives may be se- 
quenced or stepped through using the executive in oper- 
ator interface 12 or in remote task execution system 14. 

A simple preprogrammed task to change the angular 
position of an end effector or other robot device may be 
configured as a task primitive. For example, it would be 
well within the skill of a person of ordinary skill in this 
art to prepare a program, or task primitive, to adjust the 
angular position of video camera 22 in accordance with 
the absolute or relative value of an angular position 
provided by the operator. During execution of this task 
primitive by telerobot control system 10 , the operator 
would initiate this task and provide the required angular 
position information via execution terminal 32 to com- 
munication and control system processor 24. 

Video camera 22 would then be caused to change 
position in accordance with the task primitive in com- 
munication and control system processor 24 to the an- 
gular position specified by the operator input. Informa- 
tion concerning the execution of this task, as well as 
information concerning the position of video camera 22 
as detected by appropriate sensors, is then made avail- 
able to the operator via execution terminal 32. 

Telerobot control system 10 is operable in a shared 
control mode in which teleoperated inputs are merged 
during task execution with inputs from autonomous task 
primitives. Execution of shared control is initiated by 
the operator by provision of the appropriate parameters 
to communication and control system processor 24 via 
keyboard entry device 46 and/or hand controller 48. 
The preprogrammed task primitive within communica- 
tion and control system processor 24 is then provided 
with additional inputs from the operator during execu- 
tion of the task. 

An example of shared control is the task of cleaning 
or polishing a surface where the robot manipulator 
manipulates a cleaning or polishing tool over the sur- 
face. Although the operator may be required to make 
decisions concerning where and when to move the 
polishing tool along the surface, it is important to con- 
trol the force applied by the tool against the surface 
even if improperly specified by the operator. 

Under shared control, the location of the polishing 
tool on the surface, that is, along the X and Y axes or 
degrees of freedom (DOF’s) which define the plane 
tangent to the surface, as well as rotation of the tool 
about an axis perpendicular to the surface, is specified 
by the operator by teleoperation. The Z axis or DOF 
tangent to the surface, as well as tilting of the polishing 
tool, is under autonomous control to automatically 
maintain the preselected force applied by the tool and 
maintain the tool tangent to the surface. In this manner, 
the polishing may be safely accomplished by the opera- 
tor and any accidental or mistaken intended motion 
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indicated by the teleoperator along the Z axis, or tilting 
motion of the XY plane, which might apply sufficient 
forces to damage the surface, will be ignored by the 
autonomous system controlling the actual robot motion. 

Another example of shared control is the teleopera- 
tion of robot arm 20 to move toward an object during 
operation of a task primitive providing guarded motion. 
A guarded motion primitive permits motion until a 
predetermined limit is reached, such as the surface of 
the object towards which robot arm 20 is moved. A 
position sensor, not shown, would provide position 
information to communication and control system pro- 
cessor 24 indicative of the relative position between 
robot arm 20 and the surface limit. 

Upon initiation of such shared control operation, the 
operator parameterizes the task primitive by providing 
parameter inputs thereto, including the selection of the 
surface of the object as the limit and a safety zone or 
guard band, if required. Actual motion of robot arm 20 
toward the object is initiated and maintained by the 
operator via activation of hand controller 48. If the 
motion indicated by such activation of hand controller 
48 moves robot arm 20 so that its position approaches 
the surface, or guard band, provided by the parameter- 
ized task primitive, autonomous control in accordance 
with the task primitive is merged with teleoperation to 
stop motion of robot arm 20 at the preselected position 
limit. 

Task primitives under supervisory and/or shared 
control may be selected in a predetermined sequence to 
create more complex tasks. A guarded motion primitive 
may conveniently be followed by a compliant motion 
primitive controlling the grasp of end effector 21 
mounted on robot arm 20 to create a more complex 
task, called reach and grasp herein, during execution of 
which robot arm 20 is moved to the surface of an object 
which is then grasped by end effector 21. 

A compliant motion primitive for controlling the 
grasp of end effector 21 could, for example, be used to 
adjust the position of end effector 21 to control the 
contact forces while closing the end effector gripper 
fingers. Sensory feedback provides information con- 
cerning the contact forces and torques while parameter- 
ization of the compliant motion task primitive sets the 
control parameters. 

Supervisory control to move robot manipulator 16 to 
reach and grasp an object in accordance with this se- 
quence requires selection of the individual task primi- 
tives, their sequencing, and parameterization of each 
such task. Shared control while executing this group of 
tasks permits the addition of further teleoperation input 
by the operator during execution of the task. 

Shared control with task primitive sequencing per- 
mits the execution of complex tasks, such as reach and 
grasp, in response to unplanned scenarios and/or in 
poorly modeled environments, based on combinations 
of teleoperation with a group of preprogrammed, gen- 
eralized task primitives, such as guarded and compliant 
motion. 

Telerobot control system 10 is specifically configured 
for the convenient development, testing, and execution 
of complex tasks under supervisory and/or shared con- 
trol based on a preprogrammed set of less complex task 
primitives. 

The development of task primitives, and/or the de- 
velopment of more complex tasks based on sequencing 
of less complex task primitives, will be described next 
with reference to setup terminal 28, shown in FIG. 1. 
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As noted above, setup terminal 28 may conveniently be 
combined with execution terminal 32 and, for the pur- 
poses of this description of the development of a com- 
plex task, will be considered to be identical therewith. 
5 Such terminals are provided with access to a set of task 
primitives equal to, or at least related to, those con- 
tained in remote task execution system 14. 

Development of a complex task from the sequencing 
of a series of less complex task primitives may be ac- 
10 complished in telerobot control system 10 interactively, 
using either teleoperation and/or editing to provide 
appropriate parameterization. 

In the simplest case, the editing mode may be used to 
create or modify a script for the complex reach and 
15 grasp task by combining the individual movement and 
grasping task primitives described above, with appro- 
priate parameterization, to create the equivalent of a 
new preprogrammed task. 

In this case, the operator using setup terminal 28 
20 interactively enters data via keyboard entry device 56 
to combine portions of preexisting text within central 
processor 54, as displayed on monitor 52, and/or mod- 
ify default and/or previously modified parameters. 

Alternatively, the reach and grasp task may be devel- 
25 oped for a particular scenario by modifying parameter- 
ization by teleoperation of local robot manipulator 40 
and/or local graphics simulator 38 by setup terminal 28. 
Central processor 54 must of course include at least 
portions of the task primitives to be used, such as 
30 guarded and compliant motion. 

In the preferred embodiment of the present invention, 
adjustment or modification of the default or previously 
modified parameterization of these primitives is accom- 
plished interactively by the operator by both entry of 
35 data via keyboard entry device 56 and teleoperation via 
hand controller 48 as well as autonomously based on 
task context by, for example, run-time binding as will be 
discussed below in greater detail. Feedback of the oper- 
ation of local robot manipulator 40 in response to the 
40 developed task may be provided by the appropriate 
sensors, similar to force-torque sensor 19 and associated 
with local robot manipulator 40 and/or setUD terminal 
28. 

The development of such complex tasks by sequenc- 
45 ing and altering of task primitives combined with man- 
ual teleoperation may be substantially enhanced by 
combining the text mode with a learning mode. A de- 
velopment of a complex task of this type will be de- 
picted with regard to simulation terminal 30 which is 
50 used to create a simulation of a task based on a new 
scenario. For safety, convenience, speed, and/or similar 
requirements, this complex task is first simulated on 
graphics simulator 38 which provides a computer aided 
graphical simulation of the expected operation of robot 
55 manipulator 16. The task may then be simulated on local 
robot manipulator 40. 

During operation of simulation terminal 32, previ- 
ously prepared textual development of program tasks or 
sequences of task primitives are combined with teleop- 
60 eration to define a more complex task. As a simple ex- 
ample, the operator may choose to have simulation 
terminal 30 operate in a learning mode to remember a 
starting position for robot arm 20 as a preliminary step 
before execution of the reach and grasp task compiled 
65 from a sequence of preprogrammed task primitives. The 
remembered combination of the starting position, reach 
and grasp then becomes a new task primitive or macro 
which itself is then simulated and tested using graphics 
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simulator 38 and/or local robot manipulator 40 and 
further developed in the textual and learning mode by 
the operator. 

Another example is where the operator can specify a 
trajectory for robot manipulator 16 by teleoperation. 
The trajectory is then stored as parameters for a partic- 
ular motion. 

A substantial benefit of the flexibility of further devel- 
opment provided by telerobot control system 10 is that 
a relatively small set of preprogrammed task primitives, 
fully tested and qualified for use with remote task exe- 
cution system 14, can later be utilized as necessary by 
altering the parameterization of the task primitives to 
meet the robot control requirements of a new poorly 
modeled environment and/or unplanned scenarios. The 
developments of such new instantiations of the prepro- 
grammed task primitives, that is, task primitives with 
new parameterization, can be performed without risk 
or unreasonable burden of prior training on the part of 
the operator. 

In a typical use of telerobot control system 10, an 
operator is first trained on setup terminal 28 and then on 
simulation terminal 30 to perform teleoperation and 
parameterization of primitives for supervisory control, 
and then shared control using simulation by graphics 
simulator 38 as well as observation of local robot manip- 
ulator 40. New tasks, and/or modified tasks, may then 
be developed by such an operator without risk to the 
devices, such as remote task execution system 14, in the 
remote environment. Substantial testing of such new 
tasks or modification are thereby performed before 
actual operation of remote task execution system 14. 

The result is that new tasks may be performed based 
on local development and testing of combinations of 
previous tasks, task primitives and teleoperation. In this 
way, remote task execution system 14 may be safely and 
successfully operated in a poorly modeled environment 
and/or to perform tasks required by unplanned scenar- 
ios, without the requirement that untested program- 
ming be used first upon remote task execution system 
14. 

Remote task execution system 14 includes communi- 
cation and control system processor 24 to control robot 
manipulator 16 and provide a family of task execution 
primitives as well as an executive program resident 
therein. In space operations, for example, it is very 
important that the set of primitives in the remote loca- 
tion, such as a space vehicle, have been fully tested and 
are capable of being combined to perform all the tasks, 
planned or unplanned, before the original space vehicle 
launch. 

While the particular family of task primitives will 
vary depending upon the application intended for each 
particular system, a family of such primitives will be 
described generally in order to facilitate an understand- 
ing of the present invention. I addition to the family of 
task primitives and executive task sequencing, remote 
task execution system 14 provides periodic task status, 
sensor position and other feedback information as well 
as command result information to operator interface 12 
via communication link 18 and operator interface com- 
munication processor 34. 

As described above, the task primitives include 
guarded and compliant motion primitives. The guarded 
motion primitives are provided in two forms. 

The first form is a Cartesian guarded motion primi- 
tive which provides the ability to control motions using 
Cartesian coordinates including interpolated arm mo- 
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tion and stop on destination position or sensed force or 
torque thresholds. The inputs appropriate for this primi- 
tive include the designation of the specific robot manip- 
ulator to be used, selection of time or velocity as a basis 
5 for controlling motion and the magnitude thereof, force 
and torque thresholds, coordinate frames in which colli- 
sion forces are to be sensed and in which Cartesian 
interpolation is to be performed, position destinations to 
go through and the period or rate at which status re- 
1° ports are to be provided to operator interface 12. 

The second form for the guarded motion primitive is 
the joint guarded motion primitive in which joint inter- 
polation is used instead of Cartesian interpolation. In 
this primitive, joint angles are used to specify destina- 
15 tions. 

Another major task primitive is the move-to-touch 
primitive used to specify motion of a robot actuator, 
such as robot arm 20, with Cartesian interpolated mo- 
tion until a specified destination is reached or until a 
20 force or torque threshold is exceeded. If the force or 
torque threshold is reached, then back off until below 
reverse thresholds, above safety thresholds, or return to 
initial position. Inputs include selection of the device to 
25 be actuated, the Cartesian destination, the frame in 
which to perform Cartesian interpolation, the forward, 
reverse and safety force and torque thresholds, the for- 
ward and reverse velocities and the status update rate. 

Another major primitive is the generalized-compli- 
30 ant-motion or GCM primitive used for performing 
compliant motion tasks in Cartesian space. The required 
inputs include the selection of the robot actuator, the 
coordinate frame to be used for the destination, the 
frame to be used for interpolated motion, the frame to 
3 5 be used for control, selection of time or velocity based 
motion, selection of time or velocity for positional mo- 
tion, position-force selection vector to select position 
and force degrees of freedom (DOF's) in the control 
frame, compliance selection vector to select which 
4Q position DOF’s also have compliance, force-compli- 
ance control gains, gains for virtual springs, force- 
torque and position-orientation thresholds, and ending 
conditions including a selection integer selecting which 
ending conditions to test for, such as maximum errors in 
45 position, orientation, force and torque and their rates of 
change. 

The compliant grasp primitive closes the gripper 
fingers of end effector 21 while performing force con- 
trol to control contact forces. Inputs include which 
50 actuator to be selected, gripper type such as pneumatic 
or served, selection of frame in which to do force con- 
trol, force control gains and force control setpoints, and 
force-torque and position-orientation thresholds. A sim- 
ilar primitive is the free grasp primitive which simply 
55 opens or closes the gripper portion of end effector 21. 

Another major task primitive provides force reflect- 
ing teleoperation. 

Telerobot control system 10 provides a hierarchical 
menu system to guide the operator during description 
60 or development of a task from general motion types at 
the top of the hierarchy to specific motion types at the 
bottom of the hierarchy. The result is the specification 
of task primitives and their parameterization to perform 
the tasks desired by the operator. The operator need not 
65 know the specific task primitives to be used. Instead, the 
operator specifies a generic motion type, e.g. guarded 
motion, move to contact, compliant motion, force re- 
flecting teleoperation, or grasp. A new menu then is 
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provided with interaction germane to the specific mo- 
tion type. 

For example, if the operator specifies compliant mo- 
tion, the compliant motion menu may present hinge, 
slide, screw, insert, level, push, translation, and similar 5 
options. The operator’s selection of one of these options 
invokes a new menu with input selections pertaining 
only to that type of motion. The insert menu permits the 
operator to select the insertion direction, et cetera. The 
interactive hierarchical approach substantially reduces 
the number of decisions to be made by the operator at 
any one point in time while still permitting the develop- 
ment of a relatively complex, specific task. 

Complex tasks may be developed by task sequencing, 
that is, selecting a sequence of tasks to be performed 15 
sequentially. Such sequences, together with specified 
parameters, may be utilized as macros to further reduce 
the complexity of task development. For example, the 
operator can specify a door opening motion by selection 
of the appropriate door opening motion from a menu. 20 
The task as actually executed is the GCM primitive 
with specific parameterization. The operator builds the 
macro by specifying the parameters for the task primi- 
tive and then saves the macro with a unique name, such ^ 
as door-open, for future use. Several macros can be 1 
strung together to form a larger macro and saved under 
another name. A macro therefore becomes the pro- 
gramming equivalent of a command sequence. 

One convenient feature of telerobot control system ^ 
10 useful during the development of parameterized 
sequences of task primitives is the ability to set posi- 
tions, such as destinations, by use of simulation terminal 
30. A mouse, or other hand input device such as hand 
controller 68, is used to move a cursor superimposed on 35 
the simulation presented on monitor 62 or on the actual 
video presented on monitor 42. Positioning the cursor 
on the actual or simulated image of a particular spot in 
the work environment can be used as a convenient way 
to establish the position of that spot as a destination or ^ 
other parameter for a task primitive. 

Both relative and absolute motion macros are useful. 

A relative motion macro has the same relative motion 
from its starting point as when taught, or learned, even 
though the absolute starting point changes. Relative 45 
motion macros are particularly useful for repetitive 
tasks executed relative to their environment, such as 
bolt turning and grasping. Absolute motion macros 
have an absolute position destination independent of 
where they start their motion. Such macros are useful 50 
for moving to an absolute position before beginning a 
relative motion sequence. 

Relative motion macros may be executed with abso- 
lute positions, or other needed inputs, provided at the 
time of execution by means of a technique called run- 55 
time binding. Run-time binding binds parameters to the 
task execution primitives when they are being sent to be 
executed, both for direct commands and sequences. 
Some examples of run-time binding include setting the 
current reporting period parameter for the primitives 60 
and specifying the current position of video camera 22. 

Shared control has components at both operator in- 
terface 12 and remote task execution system 14 so that 
communication must be done with both by the execu- 
tive within operator interface 12 when executing shared 65 
control. The executive checks what the shared control 
mode is from the input parameters before sending the 
command to remote task execution system 14. 
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Run-time binding permits the development and use of 
a shared control macro where the mapping between the 
hand controller motion and robot motion is dependent 
on the position of video camera 22 without knowing 
where video camera 22 will be when the macro is exe- 
cuted. Run-time binding sets the actual camera arm 
position parameter when the macro is executed. 

Sequence control stops execution of a sequence if the 
command result of a command of the sequence is not 
acceptable. Each primitive has a variety of possible 
command results. When building a macro, the accept- 
able command results are specifiable. A sequence exe- 
cutes by executing each command of the sequence and 
comparing the command result with the acceptable 
command results specified in the input parameters of 
the command. The next command is executed if the 
command result from the previous command is one of 
the acceptable command results. 

Telerobot control system 10 Was configured for op- 
eration with various time delays between teleoperation 
and command inputs and task execution as would be 
encountered in certain environments, such as those 
encountered in space and undersea projects. The actual 
time delays encountered are, of course, the result of the 
actual delays present between remote task execution 
system 14 and operator interface 12 Which may change 
at unknown rates, To provide enhanced abilities to 
develop and simulate the operation of complex tasks in 
such environments, controllable variable delays, such as 
time delays 68 and 70, are provided in graphics simula- 
tor 38 and/or local robot manipulator 40, respectively. 

While this invention has been described with refer- 
ence to its presently preferred embodiment, its scope is 
not limited thereto. Rather, such scope is only limited 
insofar as defined by the following set of claims and 
includes all equivalents thereof. 

What is claimed is: 

1. A method of controlling a telerobot, comprising 
the steps of: 

providing a set of autonomous local task primitives to 
control a local telerobot simulator, each of said 
autonomous local task primitives being a reusable, 
self contained program for controlling said local 
telerobot simulator to accomplish a local subtask in 
response to the numerical values of a series of local 
input parameters related to said autonomous local 
task primitive; 

providing a related set of autonomous remote task 
primitives to control a remote telerobot manipula- 
tor, each of said autonomous remote task primitives 
being a reusable, self contained program for con- 
trolling said remote telerobot manipulator to ac- 
complish a remote subtasks, related to a local sub- 
tasks, in response to the numerical values of a series 
of remote input parameters related to said autono- 
mous remote task primitive; 

interactively developing and modifying said series of 
numerical values for said local input parameters for 
each of local task primitives on the local telerobot 
simulator, to simulate the performance on the local 
telerobot simulator of a desired series of subtasks 
by the remote telerobot manipulator; 

transferring a series of numerical values for said re- 
mote input parameters, and an order of execution 
of a sequence of autonomous remote task primi- 
tives, to said remote telerobot manipulator related 
to said interactively developed and modified nu- 
merical values and order of execution; and then 
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executing a sequence of said remote task primitives in 
accordance with said transferred numerical values 
and order of execution to control the remote 
telerobot manipulator to perform said desired se- 
ries of subtasks, whereby said remote telerobot 
manipulator may be controlled to perform a de- 
sired task consisting of said series of subtasks by the 
transfer of numerical values and an order of execu- 
tion of said remote task primitives. 

2. The method of claim 1, wherein the step of execut- 
ing the sequence to control the remote telerobot manip- 
ulator further comprises: 

modifying such execution by teleoperation to simulta- 
neously merge automatic and manual control of the 15 
remote telerobot. 

3. The method of claim 1, wherein the step of interac- 
tively developing and modifying further comprises the 
steps of: 

interactively modifying and testing the numerical 
values for the local input parameters and the order 
of execution on a local graphics simulator; and then 

interactively modifying and testing the numerical 
values for the local input parameters and the order 
of execution on a local robot manipulator. 

4 . The method of claim 1, wherein the step of interac- 
tively developing and modifying further comprises the 
step of: 

providing a time delay between task command and 
execution on the local robot simulator representing 
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the delay expected between task command and 
execution on the remote telerobot manipulator. 

5. The method of claim 1, wherein the step of interac- 
tively developing and modifying further comprises the 

5 steps of: 

combining teleoperation of the local telerobot simula- 
tor with execution of a subset of said local task 
primitives on the local telerobot simulator to de- 
velop a macro task; 

10 utilizing teleoperation of the local site simulator to 
indicate task primitives, their sequence, and their 
numerical parameterization to develop a macro 
task; 

testing, editing and retesting the macro task on the 
N local telerobot simulator; and 
then combining the pretested macro task with other 
local task primitives to develop the series of numer- 
ical values and order of execution. 

6. The method of claim 1, wherein the step of execut- 
20 mg the sequence of said remote task primitives to con- 
trol the remote telerobot manipulator further comprises 
the step of: 

modifying the parameters during execution. 

7. The method of claim 1, wherein the step of execut- 
25 ing the sequence of said remote task primitives to con- 
trol the remote telerobot manipulator further comprises 
the step of: 

modifying the parameters at the initiation of the exe- 
cution of a remote task primitive based on task 
30 context at the time of said execution. 

***** 
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